Fix running an output plugin with :onyx.core/params longer than 0 #894
Conversation
When :onyx.core/params is set, the (identity) function in operations.cljc throws because it can't handle more than one argument. Instead of usinging identity, we return the last argument (the segment).
|
So the use case you're trying to accomodate is that you can pass the actual connection as an argument to the output task, rather than the details how it can establish the connection? Is there any reason why you're not using the "regular" pattern of opening a connection upon construction of the output plugin, e.g. in your situation here https://github.com/arctype-co/onyx-mongo/blob/master/src/onyx/plugin/mongo.clj#L129 ? |
|
Hello Leon @solatis, I first developed the plugin using that pattern, which I will probably go back to for the output instead of requiring the lifecycle hooks because it is simpler to program for the end user that way. The lifecycle hook adding the function params is for supporting reads/queries in :onyx/type :function tasks, so the same lifecycle hooks can be used in either context. Regardless however of my design choices there, I still think this is a bug, because if I for example added a lifecycle hook with some :onyx.core/params targeting :all tasks, which would be a more valid use case, it would also raise this exception. |
|
Ok, I think accepting this patch would be fine, but I wouldn't mind the input of @lbradstreet on this -- perhaps there was a better reason why these params only accept a single arg function. |
|
I'm not sure I completely understand the use case, but the way I would think you would make this work is you would check whether it's an output task in https://github.com/arctype-co/onyx-mongo/blob/master/src/onyx/plugin/mongo.clj#L90 (via inspecting the task map), and you would just not inject into Ignoring the params as you've done also works, but I think you're more likely to code something which drops params silently. It's probably fine though. |
When :onyx.core/params is set, the (identity) function in
operations.cljc throws because it can't handle more than one argument.
Instead of usinging identity, we return the last argument (the segment).
This is sort of an unusual thing to do, to attach params to a lifecycle affecting an output task, but it is for a 'generic' set of lifecycle calls in my use case in the mongodb plugin I am developing. We provide the :mongo connection as a parameter so you can do queries in normal :function tasks, and use the same lifecycles for the :output tasks to open/close the connection.